On recommending

The core of any document retrieval system is a mechanism that ranks the documents in a large collection in order of the likelihood with which they match the preferences of any person who interacts with the system. Given a broader interpretation of “recommending” than is commonly accepted, such a preference ordering may be viewed as a recommendation, made by the system to the information-seeker, that is itself typically derived through synthesis of multiple preference orderings expressed as recommendations by indexers, information-seekers, and document authors. The ERIn (Evaluation--Recommendation--Information) model, a decision-theoretic framework for understanding information-related activity, highlights the centrality of recommending in the document retrieval process, and may be used to clarify the respects in which indexing, rating, and citation may be considered analogous, as well as to make explicit the points at which content-based, collaboration-based, and context-based flavors of document retrieval systems vary. On Recommending Furner 2 Introduction and Overview Since the early 1990s, the act of recommending (i.e., of making a recommendation) has become increasingly of interest to members of the information studies community, and especially to designers of document retrieval systems and digital libraries. In general, the core of any recommender system is a mechanism that carries out analysis of a large collection of real-life objects, so that the objects in the collection may be ranked in order of the likelihood with which they match the preferences of any person who interacts with the system. Typically, the highest-ranked objects will be presented as recommendations or suggestions to the user. Such recommendations may be viewed as predictions of the user’s preferences. Many recommender systems currently incorporate mechanisms for ranking documents on the basis of the extent to which they have received recommendations in the course of prior system usage, and information-seekers are thus given the opportunity to benefit from a form of social feedback or indirect collaboration. Although the literature on specific implementations of recommendation-based retrieval mechanisms has proliferated, extended explorations of the concept of recommending are rare. In this paper, a model of information-related activity is proposed in which the act of recommending is assigned a privileged position. It is argued that both indexing and citation can usefully be viewed as forms of recommendation, and that hybrid systems integrating content-based, collaboration-based, and contextbased approaches to retrieval may consequently be promoted as recommender systems par excellence. In succeeding sections, then, a model of information-related activity that borrows from microeconomic theory is presented as that which underlies current conceptions of recommender systems; a clarification is provided of the dimensions on which such systems may be distinguished from their content-based counterparts; the analogy between recommendation, citation, and indexing is developed; and conclusions are drawn as to the implications of this exercise for design practice. Firstly, however, a prefatory note on the terminology and literature of the field is required. A Preliminary Note on Terminology The application to systems of the term “recommenders” was introduced in an electronic mail message by Varian (1996), as a “better name than ‘collaborative filtering’” for the “general category of interest” to attendees at a workshop held a few days earlier in Berkeley, CA, in March 1996 (see Table 1). The widespread usage of the term “recommender system” appears to date from the publication in 1997 of a special issue of Communications of the ACM (CACM), edited by Resnick and Varian (1997), with that topic as its theme. Prior to then, systems of the kind to which the term refers were often characterized as “collaborative filtering” systems, following the usage established by Goldberg et al. (1992) in their report of Tapestry, a program that allowed users to filter incoming streams of electronic mail messages in novel ways. Following the 1996 event in Berkeley, a succession of workshops on collaborative filtering or recommender systems have been held: in Budapest, Hungary (November 1997); Madison, WI (July 1998); Pittsburgh, PA (May 1999); Stockholm, Sweden (August 1999); Berkeley, CA (August 1999); Dublin, Ireland (June 2001); and New Orleans, LA (September 2001; see Table 1). Papers reporting on recommender-system research are also regularly contributed to the conferences of the Association for Computing Machinery’s Special Interest Groups on computer--human interaction (SIGCHI), information retrieval (SIGIR), and electronic commerce (SIGecom), inter alia. Readers seeking to understand current issues and trends in the field are referred to the latest proceedings in these series of workshops and conferences. Influential accounts of system design include those given by Hill, Stead, Rosenstein, and Furnas (1995), Shardanand & Maes (1995), and Konstan, Miller, Maltz, Herlocker, Gordon, and Riedl (1997); Pazzani (1999), Paepcke, Garcia-Molina, Rodriguez-Mula, and Chu (2000), and Schafer, Konstan and Riedl (2001) provide helpful overviews. There are two separate senses in which we may classify systems as recommender systems. In one (less common) sense, we may ascribe the responsibility for recommending to the system itself; it is the function of the system to provide recommendations for its individual users as to the order in which the On Recommending Furner 3 objects in a collection should be considered. In another (more common) sense, the recommending is done by the users of the system; it is the function of the system to synthesize multiple users’ recommendations of objects in the form of a single ranking for the individual user. This distinction may appear trivial, but it should be clear that the class of recommender systems is far more inclusive if we accept the first of these two readings. For in that case any retrieval system that implements a mechanism by which the objects in a collection are ranked in order of their predicted utility for the searcher may be regarded as a recommender system; the set of objects retrieved comprise the recommendation---in other words, the suggestion or the prediction---made by the system. “Recommender system” may thus be viewed as a synonym for “retrieval system,” and document retrieval (or document recommendation) systems defined, on the basis of the type of objects being retrieved or recommended, as systems that recommend documents or document-like objects to the searcher. On the other hand, if we accept the second of the two readings, the class of recommender systems may be construed as a subset of the class of retrieval systems. Recommender systems are then those retrieval systems that effect retrieval specifically through analysis of the judgments of utility made by previous users, rather than or in addition to (for instance) direct analysis of the contents of items (as in content-based systems), or analysis of the relationships perceived to exist between items (as in contextbased systems; Furner, in review). However, even if we do accept the second reading, we may still wish to argue that all retrieval systems are recommender systems (as in the first reading) by contending that the assignment of documents to terms (indexing) that goes on in content-based systems, and the linking of documents to one another (citing) that goes on in context-based systems, are both instances of recommending activity by which preference orderings are expressed in the form of judgments of utility, relevance, relatedness, or approval. Indeed, this is the argument that is made in this paper. Information-Related Activity: The ERIn Model In this section, I present a descriptive model of certain kinds of information-related activity. It is intended that the simplifications and emphases on which this model is based will be helpful ones to adopt in the framing of any subsequent analysis of the motivations for, influences on, and products of such activity. It is also intended that the particular analogies and distinctions that are suggested by this model will inspire designers of operational information systems to develop techniques and features that support information-related activity more successfully. In subsequent sections, some of these analogies and distinctions are described in detail; in particular, the act of recommending is identified as an operation that is fundamental all information systems, and some implications of such a conceptualization for design practice are highlighted. The base of the model is a distinction drawn between three types of information-related activity, as follows: [A] the act of determining the value, worth, or virtue of engaging in the examination of a given document; [B] the act of expressing an observable statement of the perceived worth of a given document; and [C] the act of informing oneself by examining the content of a given document. We may call these categories of activity [A] Evaluative, [B] Recommending, and [C] Informative activity (ERIn), respectively. Adoption of this schema allows us conveniently to consider separately the tasks of [A] judging the “relevance” of documents, [B] “retrieving” them, and [C] “reading” them; and to highlight the analogies that may be drawn between, for instance, the actions of retrieving, recommending, citing, linking, and indexing as members of category B. Before examining the details of the model, it may be helpful to sketch, as an example, an outline of a familiar scenario to which the model may be applied. Let us imagine that the elements of this scenario are as follows: On Recommending Furner 4 1. an information-seeker---i.e., a person who desires to inform herself in some way (we might say that she desires to resolve some part of her “anomalous state of knowledge,” or to “make sense” of some part of her world; we might also characterize this desire as an information need; Dervin & Nilan, 1986); 2. a collection of documents, each of which may at any given time be judged by the information-seeker to be related to her information need in some way and to some extent (again, we might say that the documents vary in the extent to which they are capable of satisfying that need, or in the degree to which---in the given context or situation---they are relevant, useful, valuable, or worthy of examination; Schamber, 1994); and 3. an information system whose function is to enable the information-seeker to locate and read those documents that she would judge to be most closely related to her need if (which is usually not the case) she were able and willing to examine all the available documents in the collection. A familiar sequence of events that occurs when an information-seeker interacts with an information system runs as follows: (1) the prediction, by the information-seeker, that documents having a certain property or set of properties (such as their coverage of a certain topic or subject) would be relevant to her; (2) the specification, by the information-seeker, of a query expressing her prediction; (3) the determination, by the system, of the degree to which every document in the collection has the property specified in the query; (4) the presentation, by the system, of a ranked list of pointers to (surrogates for, or representations of) the n documents it judges thus to be most “similar” to the query; (5) the revision, by the information-seeker, of her initial prediction, in the light of the new evidence supplied in the form of the list of recommended documents; (6) the expression, by the information-seeker, of a modified or expanded query, reformulated to reflect her revised prediction; repetition of steps (3), (4), and (5); (7) the examination, by the information-seeker, of those documents that she predicts to be most relevant. The mere specification of such a sequence of events is, of course, nothing new; rough approximations to the sequence specified here have been common starting points for many of the sophisticated models of retrieval developed in the literature of information retrieval (IR) since the 1950s (see, for example, Maron & Kuhns, 1960; Robertson, 1977; and Salton & McGill, 1983). The distinctive clarification made in the present paper is that, in this sequence of events, the actions taken in steps (1), (3), and (5) may be conceived as evaluative acts of Type [A]; those in steps (2), (4), and (6) as recommending acts of Type [B]; and that in step (7) as an informative act of Type [C]. There are many other kinds of information-related act that may be identified as exemplifying one or other of the three types [A], [B], and [C]; there are many other kinds of sequence of informationrelated events that may be recognized as cycles of acts of these types. One corollary of the argument developed in this paper is that other sequences of information-related events---no less familiar than the one described above, but typically distinguished from it as instances not of information-seeking but variously of recommending, citing, indexing, etc.---may be modeled in terms of the same basic categories of activity employed here. One product of activity of any of these kinds (although of informative activity it happens to be only a secondary product) is assumed to be a preference ordering over a set of options (Wong & Yao, 1990; Yao, 1995; Cohen, Schapire, & Singer, 1999). The concept of preference is central to this model, as it is to models of consumer behavior in the theory of microeconomics (see, for example, Kreps, 1990; Hausman, 1992). Equally fundamentally, activity of all three kinds is treated as activity in which decisions are made. Again, it is in economic theory---and in related sub-fields of economics, statistics, and philosophy, such as decision theory (Luce & Raiffa, 1957; Resnik, 1987), utility theory (Fishburn, 1970), game theory (Von Neumann & Morgenstern, 1944), and social choice theory (Arrow, 1951; Johnson, 1998)---that decision-making activity is typically characterized as activity in which individuals, with given beliefs, seek to satisfy whatever preferences they have for the consequences of their actions. It might be considered that the ERIn model bears some affinity to rational choice (RC) theory (Becker, 1976), since one of the important simplifying assumptions that are made is that people generally On Recommending Furner 5 act “rationally,” where rationality is equated with the instrumental intention to maximize the extent to which their personal preferences are satisfied. Indeed, RC theory is commonly conceived as an approach to the explanation of social behavior that rests on this very assumption, already made explicit in microeconomics and the related fields mentioned above. Although RC theory employs the concepts of rationality and utility in much the same way as, for example, decision theory, it is distinct from the latter in that its (non-normative) aim is to explain and predict the behavior of groups of people; decision theory, in contrast, aims (normatively) to demonstrate how individuals ought to act if they were rational. Another common thread in the following discussion is the general analogy that may be drawn between document retrieval systems and voting systems (Pennock, Horvitz, and Giles, 2000). Just as a voting system allows a community of voters to make a combined recommendation to policy-makers as to the selection of policies or office-holders (see, for example, Dummett, 1984), it will be shown that a retrieval system correspondingly allows a community of judges of document relevance to pool their resources in providing guidance to information-seekers making decisions to view or to read. The relationships between the three kinds of information-related activity and their products are depicted in Figure 1; the rest of this section fleshes out the bare bones presented in that diagram. The RC-Theoretic Foundations In the rationalist models of decision theory, whose terminology is summarized in Table 2, individual people may be viewed as agents or actors (labeled [1] in the Table) in the essential sense that they voluntarily engage in interaction with the physical world and with other agents. Many of the actions [2] of humans may be characterized as goal-directed or instrumental, in that we may say that they are carried out in order to bring about some intended change in the prevailing state of affairs. The new state of affairs that is the intended outcome (consequence, result) [4] of such an action may be considered the goal of that action. On any given occasion, in any given situation, an agent may choose (i.e., make a selection) from a range of potential actions or courses of action. The members of this set of potential actions may thus be referred to as options or alternatives [3], and the decision that is made when resolving to act in a selected manner may be viewed as a choice among the available alternatives. An agent comes to such a decision through consideration of a set of beliefs [5], primarily about the nature of the predicted outcome of each available option, but also about the relatedness of particular options and particular outcomes, and the relative significance, priority, or relevance, of the different criteria on whose basis competing outcomes are evaluated. Beliefs of the first kind may be expressed as beliefs about the truth of propositions about outcomes; these propositions may in turn be expressed as object--attribute--value triples, where the outcomes are the objects, and the attribute-types may be viewed as the criteria [6] upon which decisions are made and actions are chosen. Each instance of a belief may thus be viewed as an agent--outcome--criterion--value quadruple. For example: Suppose I (the agent, [1]) am at home, it is now 8am on September 24, and I have to attend a meeting at UCLA at 8.30am. There are a variety of routes (i.e., options, [3]) I can take on my way to work, and I need to choose among them (this is the action, [2]). The time that I arrive at work (i.e., the outcome, [4]) will depend on the route that I choose. Let’s say that if I take Sunset Boulevard, for instance, I can predict that I will arrive at the office at 8.45; if I take Beverly, on the other hand, I will arrive at 8.25. Clearly, I would prefer to arrive at 8.25 rather than at 8.45: in other words, my belief [5] is that, on the basis of a criterion [6] like usefulness (in allowing me to meet my goal of arriving in time for the meeting), the former outcome has a greater value for me than the latter. My belief may thus be expressed in the form of the quadruple “Jonathan -arrival at 8.25 on September 24 -usefulness -1.” Stated in this way, the example is one of decision-making under certainty, where we are certain that the choice of option x will have the outcome y. Choice theorists are also interested in decisionmaking under risk---where any given option x may have any of a range of outcomes of known probabilities of occurrence---and decision-making under uncertainty---where the probabilities of occurrence of possible outcomes of option x are not known. To vary the circumstances of the example slightly, we might imagine---more realistically---that there are in fact several possible outcomes of traveling to work either via Sunset or via Beverly, whose occurrence depends on the level of traffic On Recommending Furner 6 encountered. Indeed, in certain circumstances (if there happens to be construction work on Santa Monica, for instance), it might turn out to be faster to travel via Sunset. In this case of decision-making under risk, the beliefs [5] that will need to be taken into account if we are to determine whether my choice is a rational one include not only my beliefs about the value to me of certain outcomes, but also about the probabilities with which those outcomes will occur. When we can not be certain about what the outcomes of given options will be, the decision-making situation may be viewed as if it were a lottery, with outcomes as prizes. As we shall see, document evaluation may be viewed as an instance of this kind of decision-making under uncertainty. It is assumed that consideration of her beliefs will lead the agent to identify her preferred option--the one whose predicted outcome she believes to be “the best” (or in other words, the one whose predicted outcome she likes or desires the most). By extension, in any case in which there are two or more available options, the agent will have a preference ordering---a ranking of all options in order of their predicted outcome’s degree of desirability, acceptability, utility, value, or worth for the agent, or the level of approval that the agent has for it. As a concept inherited from the nineteenth-century utilitarian philosophers, “utility” has often been used in the economic literature to refer to that generic criterion or property of any given agent--outcome pair on which the outcome’s position in the agent’s personal preference ordering is based. The agent’s motivation to act is assumed to be a general desire to effect the desired change in the prevailing state of affairs, and in so doing to achieve her goal by satisfying her preferences---i.e., by ensuring that her outcome of choice is the one that actually occurs. The existence of factors outside the control of the individual agent, such as the actions of others, may mean that the agent finds it impossible to control events in precisely the manner that she desires, with consequences that are less acceptable. Nevertheless, it is the rationalist assumption of this model that individual agents invariably act in order to maximize the probability that their preferred outcomes prevail. Preference orderings may be classified on at least three dimensions, as follows: 1. Level of measurement. If an absolute value, acting as a measure of the degree to which an option is approved, is (or can be) assigned to each option, and a preference ordering is derived from inspection of those values, then we may speak of a interval ordering, and (for whatever it may be worth) reliably calculate absolute differences in degree of approval. If, on the other hand, our knowledge is simply of propositions of the form “A is preferred to B” (rather than, e.g., “A is x, B is x-1”), we have a purely ordinal ranking. 2. Ordinal number. If there are only two possible ranks in the ordering (e.g., preferred and nonpreferred, approved and non-approved, top and bottom, 1 and 0), we may speak of a binary ordering. If, moreover, there is a requirement that only one option be assigned to the higher of the two ranks, we have a special binary ordering. If there are more than two possible ranks, we have a non-binary ordering. 3. Completeness. If all options in a given set have been evaluated and ordered, we may speak of a complete ordering. If some of the options in the set remain outside the ordering, yet or never to be evaluated, we have an incomplete ordering. In the following three sub-sections, we shall consider in turn how each of the three kinds of information-related activity in our model may be interpreted in the terms of the decision-theoretic framework outlined above. Evaluative Activity The first kind of information-related activity that we shall consider is (what we are calling) evaluative activity. Actions of both other kinds, recommending and informative, are dependent on preference orderings which are in turn based on certain beliefs held by the agent. Further, the agent may On Recommending Furner 7 point to evidence of various kinds in justification of her holding those beliefs. The action of weighing the available evidence in order to arrive at a preference ordering may be considered as evaluative activity (see Table 3). Taking a gross input--output view, we may characterize the input to such activity as a set of beliefs about the options under consideration and (perhaps) about the preference orderings supplied by other agents. The output is a personal preference ordering, which in turn directs the course of subsequent recommending or informative activity. Formally, to engage in evaluative activity ([2] in Table 3) is for an individual agent [1] to come to hold a set of beliefs [5]---beliefs about the likelihood with which each of the available options [3] is a member of a class specified by a particular agent--outcome[4]--criterion[6]--value quadruple---through consideration of the available evidence [7] that may involve a method [8] of combining evidence from multiple sources. For example: Suppose I (the agent, [1]) have a paper to write. It is to be on the topic of the cultivation of cantaloupes, a subject of which I am wholly ignorant. I turn to Google for some inspiration. Google’s database contains pointers to a very large number of documents (i.e., options, [3]), and I need to choose among them (this is the action, [2]). The extent to which I manage to resolve the anomaly in my knowledge of cantaloupes (the outcome, [4]) will depend on the documents that I choose to retrieve. It might be considered that my most important belief [5] is that, on the basis of a criterion [6] like usefulness (in allowing me to meet my goal of writing the paper), a knowledge-state in which my knowledge of cantaloupes is enhanced is of greater value than one in which my level of ignorance about musk-melons remains stable. But my decision-making in this situation is uncertain because, prior to either examining any of the documents in the database or using Google’s recommendations to guide me, I have very little knowledge of the probability with which my choice of any given document will produce the outcome that I desire. A more helpful interpretation of the beliefs on whose basis a preference ordering is constructed is that they are beliefs about the relationships that exist between the agent and each option (i.e., between the information-seeker and each document). Each such relationship is situational and contextual, in that it is unique to the given agent--option pair; and it is temporal and dynamic, in that it is subject to change over time. In fact, we might choose to define such a relationship by a triple (or even a quadruple) that adds context (or both place and time) to the representation. Moreover, the agent’s perceptions of, and beliefs about, the properties of each relationship are personal and subjective. For instance, we may say that to rank [2] a set of documents [3] is for an information-seeker, author, indexer, reviewer, or recommender/retrieval system [1] to determine the degree [5] to which each document is “relevant” [6], on the basis of the agent’s knowledge [7] of the document and of any prior judgments of relevance. The action in every case consists in evaluating, given a criterion X such as relevance, the degree to which each of the available options is X---i.e., the degree to which each option is a member of the class whose label is specified by a particular agent--outcome--criterion--value quadruple. The general criterion on which options are evaluated---“relevance”---may then be interpreted simply as the degree of relatedness of option to agent, or to a representative of the agent’s current context that we may call a query. Some instances of an agent--option relationship are more “embedded”---we might say they are more rooted, more intrinsic, more persistent, or less transitory---than others, to the extent that the agent has no need to specify a temporary self-representation or query. There is likely to be, for example, a difference between the degree of embeddedness of the relationship between a particular candidate in an election and a voter, and that of the relationship between a particular document in a collection and an information-seeker; both are subject to change over time, but the former is less likely to change with the rapidity and regularity that the latter will change in accordance both with changes in the day-to-day interests of the information-seeker and with her awareness and usage of the given document. We may refer to criteria that are weakly-embedded properties of query--document--context triples, as extrinsic---in contrast with those intrinsic criteria that are strongly-embedded properties of agent--option pairs. The evidence that the agent may point to in justification of her holding her beliefs may be said to vary in terms of the process by which such evidence is produced. Evidence of one kind is that produced On Recommending Furner 8 first-hand by direct observation, on the part of the agent, of the objects about which the beliefs are held. Evidence of a different kind, however, is produced by acceptance of the testimony of another agent, or of a synthesis of the testimony of multiple other agents. In our previous examples, the agent has been assumed to be an information-seeker. But the ERIn model is intended to cover the related activities not only of information-seekers, but also of automated retrieval mechanisms, as well as of indexers, citers, and raters of documents. In a situation in which we may identify a retrieval mechanism as the agent, the testimony on which that mechanism’s evaluation of documents is based may be the preference orderings of a single indexer or classifier, of a single citer or linker, or of a single rater (perhaps the current information-seeker, providing “relevance feedback”); alternatively, the testimony may include preference orderings of multiple indexers, citers, or raters, or of any combination of those agents engaged in the ERIn process. In these latter cases, the problem for retrieval mechanisms becomes one of appropriately combining, often using sophisticated mathematical methods, multiple sources of evidence of document relevance that are supplied in the form of preference orderings established by the evaluative activity of indexers, citers, and raters. Recommending Activity When considering information-related activity in general, we may distinguish, on the basis both of motivation and of outcome, between informative activity---as that which is undertaken as an end in itself, in order directly to satisfy a preference ordering---and recommending activity---as that which is undertaken as a means to an end, in order to record for subsequent consideration (by oneself or, more commonly, by others) an expression of a preference ordering. Activity of both kinds results in the expression of a preference ordering: in the first case, however, this expression may be regarded as implicit, since the generation of an expression of the ordering is a goal that is secondary or incidental to the satisfaction of the ordering. In Table 4, the properties of a range of recommending activities are summarized. To engage in recommending activity [2] is for an individual agent [1] to record an expression [4] of a preference ordering over the options [3] in a given set. For instance, to vote [2] is for an individual citizen (the voter) [1] to record an expression (the vote) [4] of a preference ordering over the candidates [3] on a slate. Correspondingly, to “search” [2] is for an information-seeker [1] to record an expression (the query) [4] of a preference ordering over the documents [3] in a collection; the query acts as the information-seeker’s prediction of the properties that are held by documents that she would judge relevant if she had perfect knowledge of the collection. To “retrieve” [2] is for a retrieval system [1] to present to its user a retrieval set of documents [4] expressive of a preference ordering over the documents [3] in a collection; to cite [2] is for an author (the citer) [1] to record an expression (the citation) [4] of a preference ordering over the documents [3] in a collection; to index [2] is for an indexer [1] to record an expression (in the form of the assignment of an index term) [4] of a preference ordering over the documents [3] in a collection; and so on. Of course, we do not need Table 4 to figure out that searchers giving feedback are essentially engaged in the same activity as the “experts” employed to make relevance judgments on the documents in collections used in experimental tests of retrieval systems. Equally uncontentious is the observation that citing and linking are analogous activities (Borgman & Furner, in press). Commonalities that are recognized less often, however, are those that connect recommending, retrieving, and indexing. In fact, it may be argued that, in all of these cases, any expression of a preference ordering may be viewed as a relevance judgment (whether or not it is commonly referred to as such). The citation or link between a citing or source document and a cited or target document may be viewed as a judgment, made by the citer, of the relevance of the target to the source. The index term assigned to a document may be viewed as a judgment, made by the indexer, of the relevance of the index term to the document. The rating assigned to a document may be viewed as a judgment, made by the recommender, of the relevance of the document to the recommender herself. In a similar sense, it may also be argued that, in all of these cases, a judgment of relevance is a judgment of approval (whether or not it is commonly recognized as such). All such judgments are On Recommending Furner 9 personal decisions made by individuals based on subjective beliefs about (or interpretations or perceptions of) the degree to which each object under consideration is a member of that class of objects which, in some sense that only the judge herself may confirm, are related either directly to the judge or indirectly to a query (a representative of description of the judge’s current context). The objective existence of relationships of this kind may of course be called into question by other judges; but such argument would be irrelevant since it is the existence of the subjective beliefs about the relationships that is of consequence. Even if we could prove the objective existence of relationships between documents and queries, it is not the case that such proof would logically determine our beliefs as to such existence; and relevance judgments (of all kinds---i.e., citations, assignments of